Recent research suggests that the medial temporal lobe (MTL) is involved in perception as well as in declarative memory. Amnesic patients with focal MTL lesions and semantic dementia patients showed perceptual deficits when discriminating faces and objects. Interestingly, these two patient groups showed different profiles of impairment for familiar and unfamiliar stimuli. For MTL amnesics, the use of familiar relative to unfamiliar stimuli improved discrimination performance. By contrast, patients with semantic dementia—a neurodegenerative condition associated with anterolateral temporal lobe damage—showed no such facilitation from familiar stimuli. Given that the two patient groups had highly overlapping patterns of damage to the perirhinal cortex, hippocampus, and temporal pole, the neuroanatomical substrates underlying their performance discrepancy were unclear. Here, we addressed this question with a multivariate reanalysis of the data presented by Barense et al. (2011), using functional connectivity to examine how stimulus familiarity affected the broader networks with which the perirhinal cortex, hippocampus, and temporal poles interact. In this study, healthy participants were scanned while they performed an odd-one-out perceptual task involving familiar and novel faces or objects. Seed-based analyses revealed that functional connectivity of the right perirhinal cortex and right anterior hippocampus was modulated by the degree of stimulus familiarity. For familiar relative to unfamiliar faces and objects, both right perirhinal cortex and right anterior hippocampus showed enhanced functional correlations with anterior/lateral temporal cortex, temporal pole, and medial/lateral parietal cortex. These findings suggest that in order to benefit from stimulus familiarity, it is necessary to engage not only the perirhinal cortex and hippocampus, but also a network of regions known to represent semantic information.